Policy network in chess engines
Policy network
Definition
In computer chess that uses neural networks, a policy network (often called the policy head) is the component that, given a chess position, outputs a probability distribution over all legal moves. In plain terms, it answers the question: “Which moves look most promising from here?” The output is a set of move probabilities that sum to 1, assigning higher probability to moves the network believes are strong.
The policy network is commonly paired with a value network (which estimates how favorable the position is for the side to move). Together, they guide search algorithms such as Monte Carlo Tree Search (MCTS) used by engines like Leela Chess Zero (Lc0) and AlphaZero.
How it is used in chess
Policy networks are deeply integrated into the decision-making process of modern neural engines:
- Seeding MCTS: The policy provides “priors” for each move at a node in the search tree. MCTS expands and revisits branches in proportion to both their prior and their current estimated value, allowing the engine to allocate more simulations to promising lines early.
- Move ordering: Even outside full MCTS, a learned policy can help order candidate moves so that the search encounters strong moves earlier, improving pruning efficiency.
- Time management and pruning: Moves with very low policy probability may be searched less deeply or pruned earlier, reducing the effective branching factor.
- Training loop: In self-play training, the distribution of visits that MCTS assigns to moves at a position becomes a “target” policy. The policy network learns to match that improved target, creating a feedback loop of policy improvement.
Note: Classical alpha–beta engines like Stockfish traditionally do not use a policy network. With NNUE (2020–), Stockfish incorporated a neural evaluation (value-like) network while still relying on handcrafted move ordering heuristics (transposition tables, history/killers, etc.). Experimental hybrids have explored using policy priors to improve move ordering, but this is not standard in mainstream Stockfish.
Strategic and historical significance
The emergence of policy networks in chess followed breakthroughs in deep reinforcement learning. After AlphaGo (2016) and AlphaZero (2017) showed that policy-and-value-guided MCTS could reach superhuman strength, Leela Chess Zero brought the same architecture to chess via distributed self-play.
- Strategic shaping: The policy network learns chess principles and patterns—central control, development, king safety—and encodes them statistically. This often yields human-like candidate move lists, emphasizing purpose-driven play.
- Efficiency: By directing search toward promising options, the policy network compensates for the combinatorial explosion of chess. It helps engines find deep tactical ideas (including sacrifices) by focusing compute where it matters.
- Learning dynamics: During training, small “policy mistakes” often get corrected first in the opening and early middlegame; over time, the policy learns sharper tactics and long-term plans as the value network and search improve.
Examples
Example 1: Classic opening guidance. After 1. e4 e5 2. Nf3 Nc6 3. Bb5, a trained policy network typically ranks 3...a6 highly, reflecting the main line of the Ruy Lopez. The policy probabilities give the search a head start on the most critical continuations.
Try visualizing the moment before Black’s third move:
Example 2: Sacrificial ideas. In some sharp Sicilian positions, Lc0’s policy may assign meaningful probability to thematic sacrifices like Bxh7+ or Nxe6 based on learned patterns, prompting the search to test those lines early. While the value network and the search ultimately validate the sacrifice, the policy is what ensures these ideas are explored quickly.
Example 3 (historical context): In the AlphaZero vs. Stockfish matches (2017), AlphaZero’s style—long-term pressure, pawn storms, and piece activity—was powered by a policy that favored dynamic ideas. Though the final move choices came from MCTS, the policy was essential in steering the search toward those plans.
Training and mechanics
- Input and output: The policy head consumes an encoded board (piece placements, side to move, castling rights, etc.) and outputs a probability over a fixed move space covering all legal moves (including promotions).
- Targets: During self-play, MCTS produces a visit-count distribution over moves from a position. That distribution, often softened by a temperature parameter, becomes the target the policy learns to imitate.
- Exploration: To avoid early overfitting in self-play, engines inject Dirichlet noise at the root so that the policy explores offbeat moves, broadening experience and improving robustness.
- Integration with search: Selection in MCTS balances exploitation (moves with higher value estimates) and exploration (moves with higher policy priors and uncertainty). The policy thus continually influences which branches receive more analysis.
Interesting facts and anecdotes
- Policy-only play: If you force a neural engine to pick the top policy move with minimal search, it still plays surprisingly “principled” chess in the opening, underscoring how much opening knowledge ends up encoded in the policy.
- Style imprint: Different training regimens and datasets can give policy networks subtly different “styles.” Some versions of Lc0 have been noted for favoring space and long-term king attacks, a tendency visible in their policy suggestions.
- Beyond chess: The policy/value split originated in Go engines; its success there inspired the same approach in chess, where it helped bridge the gap between brute-force techniques and pattern-based intuition.
Practical takeaways for players
- When using Lc0 or similar engines, the “policy” percentages you see for candidate moves reflect learned chess patterns; they are not final evaluations but a guide to where the engine expects good play.
- Moves with modest evaluation but very high policy may hide long-term potential that requires deeper search. Conversely, a low-policy move that evaluates well might require extra search time to confirm.